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Abstract:  A  light  stripe  vision  system  is  used  to  measure  the  location  of  poly¬ 
hedral  features  of  parts  from  a  single  frame  of  video  camera  output.  Issues  such  as 
accuracy  in  locating  the  line  segments  of  intersection  in  the  image  and  combining 
redundant  information  from  multiple  measurements  and  multiple  sources  are  ad¬ 
dressed.  In  2.5  seconds,  a  prototype  sensor  was  capable  of  locating  a  two  inch  cube 
to  an  accuracy  (one  standard  deviation)  of  .002  inches  (.055  mm)  in  translation 
and  .1  degrees  (.0015  radians)  in  rotation.  When  integrated  with  a  manipulator, 
the  system  was  capable  of  performing  high  precision  assembly  tasks. 
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1  Introduction 


Fast  and  accurate  location  of  objects  is  important  in  many  automated  industrial 
tasks  involving  inspection,  assembly  and  machining.  Such  a  measurement  capabil¬ 
ity  can  change  the  way  certain  tasks  are  currently  performed,  for  instance,  force 
controlled  motion  and  precise  part  fixturing  are  unnecessary  if  an  assembly  system 
can  align  mating  surfaces  of  parts. 

An  interest  in  automating  mechanical  assembly  tasks  prompted  a  study  of  sens¬ 
ing  systems  which  provide  computers  with  information  about  the  position  of  parts. 
The  most  critical  sensor  design  requirements  for  a  position-sensor-based  assembly 
system  are  high  speed  and  accurate  operation;  thus,  many  of  the  object  ranging 
systems  in  the  literature  [3,8,20,21,30,34]  cannot  be  used. 

The  system  described  in  this  paper  operates  in  a  relatively  highly  constrained 
industrial  environment.  We  assume  that  the  o  pnort  position  of  a  part  is  known 
approximately;  that  is,  the  expected  location  of  the  part  is  available  to  the  sensor 
before  it  makes  a  measurement.  With  this  assumption,  fairly  sparse  data  may  be 
used  to  obtain  a  relatively  precise  measurement.  A  conceptually  similar  approach 
was  taken  by  Benton  and  Waters  [5]  and  Rutkowski  et  al.  [29]  who  fit  a  model  to 
light  stripe  data.  Their  approach  depended  upon  an  iterative  technique  and  was 
somewhat  slower  and  less  accurate  than  the  technique  discussed  here,  however,  it 
could  locate  a  larger  variety  of  part  features. 

2  Sensing  Method 

A  system  capable  of  making  measurements  in  sue  degrees  of  freedom  is  required 
for  the  part  position  sensing  function.  In  addition,  a  flexible  (capable  of  measuring 
many  different  parts),  high  speed,  high  accuracy,  and  low  cost  system  is  desirable. 
Jarvis  [24]  presents  an  overview  of  a  number  of  three-dimensional  ranging  systems. 
The  light  stripe  ranging  technique  was  selected  because  it  best  fits  the  sensing 
system  requirements. 

The  light  plane  of  a  light  stripe  system  is  projected  across  one  or  more  features 
(such  as  a  corner  of  a  polyhedron  or  an  end  of  a  truncated  circular  cylinder)  of  a 
part,  Figure  1.  Data  from  an  image  taken  by  a  video  camera  positioned  at  some 
disparate  angle  with  respect  to  the  light  plane  is  processed  to  locate  the  feature. 

Triangulation  is  used  to  obtain  three-dimensional  data  from  the  two-dimensional 
sensing  element  data.  Figure  2.  A  common  light  source  for  light  stripe  systems  is 
a  laser  beam  which  has  been  passed  through  a  cylindrical  lens.  The  lens  diverges 
the  beam  in  a  direction  perpendicular  to  the  lens’  cylindrical  axis. 

The  light  plane  is  the  sole  functional  illumination  source  in  the  scene  as  viewed 
by  the  caunera.  An  isomorphism  exists  between  all  points  in  the  image  and  points 
lying  in  the  light  plane  in  three-space.  In  order  to  determine  the  transformation 


Part 


Figure  1:  Three  line  segments  generated  by  the  intersection  of  a  plane  of  light  and 
the  surfaces  of  a  polyhedral  feature  may  be  sensed  by  a  video  camera  and  used  to 
locate  a  part. 


from  the  two-dimensional  image  coordinates  to  accurate  three-dimensional  space 
coordinates,  the  system  must  be  precisely  calibrated.  During  the  calibration,  the 
values  of  three  parameters  must  be  determined  which  locate  the  light  plane  with 
respect  to  a  coordinate  frame  defined  by  the  camera  image  plane.  A  disparity  angle, 
Olpj  a  tilt  angle,  <^lp,  and  an  offset,  yt/’i  are  the  parameters  used  for  defining  the 
light  plane  location  (these  are  not  unique). 

Discussion  in  this  paper  is  limited  to  objects  whose  surfaces  are  planar  (actually 
only  those  surfaces  being  sensed  need  be  planar);  thus,  the  intersection  curves 
between  the  light  plane  and  part  surfaces  are  line  segments. 


2.1  Inverse  Projection  of  Image  Points 

The  coordinates  of  points  in  space  lying  in  the  light  plane  may  be  derived  as  a 
function  of  the  image  plane  coordinates  and  the  light  plane  and  camera  calibration 
parameters.  Some  authors  have  used  a  matrix  description  for  the  geometry  of  a  light 
stripe  system  [2,26],  Separate  equations  are  maintained  here.  The  transformation 
of  a  point  on  an  object,  (x,y,  z),  to  image  plane  coordinates,  (yr,Zf),  is  described 
by  the  perspective  projection  [35] 


yr  =  - 
z/  =  - 


X 


(1) 


We  may  obtain  the  x  coordinate  in  terms  of  the  y  and  z  coordinates  using  the  light 
plane  constraint.  f¥om  Figure  2 

I  =  Xo  +  tantfi,p(y  +  ztan^ip)  (2) 

where  zq  is  the  position  along  the  optical  axis  where  the  light  plane  crosses, 


Xq  =  —ytp  tantf^p. 

In  general  zo  <  0.  Combining  (1),  and  (2),  we  obtain 


z  = 


Xo 


1  +  tantfLp(tan^i,P^  + 
itii 


y  =  - 


Js. 


Xq 


1  +  tan  $Lp{tz.n  4>lp^  + 


(3) 

(4) 

1  +  tan  tfLp(tan  <f>Lpf^  +  '^)' 

These  equations  describe  the  position  of  a  point  in  space  (x,y,z)  as  a  function  of 
the  coordinates  of  the  corresponding  point  in  the  image  plane  [yi,zj). 


z  = 


/o 


Xo 


2.2  Calculation  of  Three-Dimensional  Vectors  from  the  Light  Stripe 
Image 

The  measurements  obtained  from  the  image  of  the  intersection  of  a  light  plane 
and  a  polyhedral  feature  is  a  set  of  line  parameters,  [m/,,',6/,,],  which  are  defined 
by  the  equation  of  the  line  (refer  to  Figure  3) 

yi  =  m/.,2/  +  6/,,.  (6) 

Using  (3),  (4),  (5),  and  we  may  obtain  expressions  for  the  vectors  along  the 

light  stripes  in  real  space,  and  vectors  from  the  origin  to  the  intersection  of  the 
light  stripes  and  the  z  =  0  plane,  6^, 

(  (mi  +  tan  <^/,p)  tan  $ip 
U  =  i  mi 

1 


(7) 


and 


1  +  -^ 

*/xi 


bi  = 


so 


ISL. 


1+-^ 


(8) 


where 


m/,.  -  ^  tan  <I>lp  tan  $ip 
1  +  ^  tan  9 IP 


mi  = 


(9) 
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Figure  4:  Light  plane  intersecting  a  polyhedral  feature  and  a  corresponding  model 
of  the  feature. 

3  Polyhedral  Feature  Localization 

Most  of  the  light  stripe  systems  in  the  literature  require  scanning  a  complete 
scene  and  processing  a  number  of  video  frames.  With  the  light  plane  positioned 
at  a  certain  orientation  with  respect  to  a  polyhedron,  three  or  more  line  segments 
comprise  the  curve  of  intersection  and  only  a  single  frame  is  necessary  to  locate  the 
part  given  its  approximate  a  priori  position.  The  following  geometric  conditions 
are  sufficient  for  three  light  plane  intersection  lines  to  locate  a  part  [32] 

•  Pairs  of  lines  must  not  be  parallel. 

•  Pairs  of  faces  in  which  lines  lie  must  not  be  parallel. 

•  The  common  normal  of  any  pair  of  lines  must  not  be  parallel  to  the  line  of  intersection 
of  any  two  faces  in  which  the  lines  lie. 

3.1  Locating  General  Polyhedral  Features 

The  position  of  a  polyhedral  feature  of  an  object  is  referenced  relative  to  a 
model  of  the  feature  in  a  base  coordinate  system,  Figure  4.  The  orientation  of  the 
feature  may  be  determined  by  the  slopes  of  three  line  segments  in  the  image.  The 
displacement  may  be  determined  from  the  results  of  the  orientation  calculation  and 


the  intercepts  of  the  three  lines;  thus,  the  orientation  calculation  may  be  decoupled 
from  the  displacement  calculation. 

3.1.1  Determining  the  Orientation 

We  wish  to  find  the  rotation  between  the  model  and  the  object.  Assume  that 
the  three  normal  vectors  for  faces  A,  B,  and  C  of  the  model  in  Figure  4  are  n^,  Rn, 
and  he  respectively.  The  normal  vectors  of  the  object  are  unknown,  but  vectors 
along  the  intersection  line  segments  on  the  object  are  known  from  the  sensed  data. 
Assuming  that  the  correspondence  between  faces  is  known  (approximate  a  prtort 
position),  the  unit  vectors  along  the  intersection  line  segments  for  faces  A,  and 
C  are  I  a,  h  respectively.  The  rotation  matrix  R  from  the  model  to  the 

object  is  found  using  quaternion  notation  [23,28,31].  We  denote  a  quaternion  by 
the  bold  face  letter,  q.  Every  quaternion  has  a  corresponding  four  vector,  q,  whose 
components  are  the  four  elements  of  the  quaternion.  A  quaternion  with  a  zero 
scalar  element  represents  a  three  vector;  thus  a  quaternion  multiplied  by  a  vector 
is  defined.  A  rigid  rotation  of  a  vector  v  is 

quq* 

where  q  is  a  unit  quaternion  and  q*  is  the  quaternion  complement  of  q. 

If  the  proper  rotation  is  found,  the  line  vectors,  /<,  will  be  perpendicular  to  the 
rotated  model  normal  vectors,  qn,q*;  that  is,  the  dot  product  will  be  zero 

ii  •  qn<q’  =0,  i  =  A,  J9,  C.  (10) 

Each  of  these  equations  are  quadratic  in  components  of  q.  Rewriting  this  in  matrix 
form 

f^Aiq  =  0,  i-A,B,C, 

1,1  =  1  (11) 

which  is  a  coupled  set  of  four  quadratic  equations.  Each  of  the  three  A,  matrices 
(4  X  4)  have  elements  of  quadratic  polynomials  in  components  of  /y  and  {j  and  k 
take  on  values  of  A,  ?  or  C).  The  components  of  q  may  be  determined  using  any  of 
a  number  of  numerical  methods.  The  rotation  matrix,  R,  may  be  calculated  from 
the  components  of  q  [31] 

9o  +  9?  -  -  9l  2(-,o,3  +  9igj)  2(,o,2  +  ,i,s) 

R  =  Hqoqs  +  9ift)  9o  -  9i  +  d  -  9l  2(-,o,i  +  ,29s) 

.  2(-,b9j  +  9i9s)  2(,o9i  +  9j9s)  9o  "  9?  '  9|  +  9s  . 

An  alternate  approach  to  locating  the  general  polyhedral  feature  is  to  use  an 
iterative  method  such  as  the  one  in  [5,29].  The  assumed  a  priori  location  of  the 
feature  provides  a  good  initial  guess  and  improves  solution  convergence. 


3.1.2  Determining  the  Displacement 

Once  ^lie  rotation  from  the  model  to  the  object  is  known,  the  normal  vectors  of 
the  o*  ^ect,  not),,,  he  calculated 

»o*y,i  =  B,  C.  (12) 


Unless  an  intersection  line  is  parallel  to  the  2  =  0  plane,  we  may  define  the  inter¬ 
cept  vector,  bi,  which  lies  in  the  2  =  0  plane  from  the  origin  to  the  intersection 
line.  The  equation  of  the  plane  which  lies  on  the  face  is  [15] 

{x  -  bi)  •  hci},i  =  0  (13) 


where  x  =  [i  y  2]^  is  the  location  of  a  point  in  space.  Solving  these  three  equations 
simultaneously  for  the  planes  on  faces  A,B,  and  C,  we  obtain  the  location  of  the 
corner 


_  PA[noij,B  X  +  PBynohj.c  x  n<,*y,^)  -t- 

Xeomtr  —  *  / «  »  v 

X  noijjC) 


where  x  denotes  the  vector  product  and  p,  is  the  perpendicular  distance  from  the 
origin  to  the  plane 


Pa  =  (^x  •  «o»y.x) 

Pb  =  (^B  •  »o»j\b) 

Pc  =  {be  •  «<>»>, c). 


(15) 


3.2  Locating  Right  Corner  Features 

The  solution  for  the  rotation  of  the  right  polyhedral  corner  feature  may  be  solved 
in  closed  form  (in  contrast  to  \ising  a  numerical  technique  for  general  polyhedra). 
We  define  a  right  corner  feature  as  the  intersection  of  three  half  spaces  (see  Figure  5) 
whose  bounding  planes  are  mutually  orthogonal.  A  right-hand  coordinate  system, 
Tp,  with  axes  xp,yF,  zp  perpendicular  to  the  bounding  planes  of  the  half  spaces  and 
origin  located  at  the  point  of  intersection  of  the  bounding  planes  defines  the  position 
of  the  feature  in  six  degrees  of  freedom.  We  wish  to  determine  the  orientation  and 
position  of  this  feature  reference  frame  with  respect  to  the  world  frame,  Jw ,  located 
at  the  camera  image  plane. 

3.2.1  Determining  the  Orientation 


The  orientation  of  the  feature  reference  frame,  Jp^  with  respect  to  the  world 
reference  frame,  is  determined  by  locating  an  intermediate  reference  frame 


associated  with  the  laser  plane,  Tt,p.  For  the  present  ainalysis,  we  assume  that 
the  light  plane  has  no  width  (two-dimensional).  We  also  assume  that  three  line 
s^pnents  (no  width)  are  generated  when  the  light  plane  intersects  the  feature.  The 
orientation  of  the  three  line  segments  are  defined  by  the  three  vectors  I  a,  Ib>  ^d 

A 

Ic  ns  shown  in  Figure  5. 

We  assume  the  ranging  sensor  has  been  accurately  calibrated  and  the  unit  vector 
normal  to  the  light  plane,  nr,p,  is  known.  The  intermediate  coordinate  system,  Tip, 
has  axes  xlp  in  direction  of  one  of  the  lines  of  intersection  (lx),  zlp  in  direction 
hf,p,  and  yi,p  in  direction  n^p  x 

The  rotational  transformation  (in  matrix  form)  from  Tiv  to  Tip  is 


^WJ^P  =  [Z"i4»  ^Lp  X  ix,  ulp] 


Tlp  may  be  located  with  respect  to  Tp  by  realizing  that  the  cross  product  of  any 

^  A  A 

two  of  the  three  vectors  lx,  Ibj  or  Ic  is  normal  to  the  light  plane  (in  the  direction 
of  Zlp)  and  lx  lies  on  the  yp,ZF  plane.  The  transformation  from  Tw  to  Tip  is* 


Rf.tp  =  [^(ai  (^^B  X  ^ix)  X  ^Ix,  ^Ib  X  ^ix] 


—  CO8  0X 

—  sin  9x 


I eo«*  ij,  iin*  +»in*  #x 

I co«*  tin*  ♦a+iin* 

—  cotin  co»$n  tinfj 


/ cc**  tji  fin*  0g  +»in*  fA 

— — 

tji  iin*  #0+«ii*  $A 
Mn  $a  co»  >  j _ 


f so«* “n* ix  \  to**  $x  »•»* ♦B+tin* 9a 


where  dx  is  the  angle  between  lx  and  zp,  emd  9b  is  the  angle  between  /p  and  yp. 
The  angles  9x  and  9b  may  be  calculated  from  the  angle  between  lx  and  Ic  {9xc) 
and  the  angle  between  lx  and  Ip  (9xb)  (these  angles  are  both  easily  obtainable  from 
the  processed  image) 


arctan  \/  — 


=  arccos 


V  tan  9xc 
cos  9xb 


1  Multiple  values  of  the  rotation  matrix,  Rf,lp,  which  correspond  to  different  light 
stripe  image  interpretations  result  from  these  equations.  The  proper  rotation  ma¬ 
trix  is  the  one  closest  to  the  expected  nominal  rotation  matrix.  The  orientation  of 
the  feature  reference  frame,  Tp,  with  respect  to  the  world  coordinate  frame,  Tw,  is: 


R-w,f  =  R-w.tpRpip  (18) 

'The  left  superscripts  denotes  that  the  vector  is  expressed  with  respect  to  the  7f  reference 
hame. 


Variable 

Deecription 

C 

Lise  target 

Image  of  £  in  the  camera’*  image  plane 

N 

Number  of  illuminated  pixel*. 

Coordinates  of  the  center  of  the  i*''  illuminated  pixel 

Coordinates  (probabilistic)  of  points  lyiiii;  on  £ . .  1 

h. 

Probabilistic  distance  iii  tiie  y  direciion  from  tlie  j 

ceiiter  of  the  illunitiiated  pixel  to 

e 

Orientation  angle  of  £  measured  counierclockwise 

from  the  2  axis 

Vo 

y  axi?  intercept  of  C 

!?i 

Least  squares  estimators  for  S  and  i/i, 

^  j. 

9, 

Estimators  for  S  and  pi.  from  discrete  data  (fi,;/,) 

Atf,  AjX, 

DifTereiice  between  estimated  line  parameters  yn) 

and  actual  parameters  y,,)  I 

Length  of  each  square  pixel 

Variance  of  »y,  for  all  1  =  1,. . .  N 

Distance  from  the  center  of  to  the  ti  axis 

measured  parallel  to  ( he  z  axis 

Table  1;  Nomenclature  for  line  parameter  error  analysis. 

3.2.2  Determining  the  Displacement 

Once  the  rotation  is  known,  the  displacement  of  the  corner  from  the  base  coor¬ 
dinate  system  is  given  by  14. 

4  Locating  Straight  Line  Features  in  Quantized  Images 

Determining  the  accuracy  of  measurements  of  polyhedral  features  is  studied  by 
addressing  the  more  fundamental  problem  of  determining  the  accuracy  of  fitting 
two-dimensional  line  segment  parameters.  The  precuion  of  a  least  squares  technique 
in  determining  the  orientation  and  position  of  thin  line  segments  in  the  presence  of 
quantization  noise  is  investigated.  The  probabilistic  technique  used  in  this  study  is 
similar  to  the  one  reported  by  Hill  [22]. 

4.1  Errors  in  Fitting  Linear  Parameters  to  Discretized  Data 

Data  generated  by  straight-line  features  appear  as  discrete  points  of  various 
intensities  located  within  some  width  of  a  central  axis  in  the  image  plane  of  a 
discrete  array  sensor.  These  points  can  be  processed  to  find  the  best  fit  line  through 
them  using  a  least  squares  technique  [1,4].  This  section  explores  the  accuracy  with 
which  straight  line  features  may  be  found.  The  variables  used  in  the  subsequent 
analysis  are  listed  in  Table  1. 


Figure  6;  Parameters  for  the  image  of  a  line  (infinitely  thin). 

4.1.1  Theoretical  Errors  from  Linear  Least  Squares  Approximations 

For  the  following  analysis  it  is  assumed  that  an  image  is  generated  from  a  single 
line  (infinitely  thin),  C,  which  lies  in  a  plane  parallel  to  the  image  plane.  The  line 
is  located  at  orientation  9  from  the  horizontal  (x)  axis  and  passes  over  the  y  axis  at 
intercept  yo.  For  simplification,  the  projection  of  C  into  the  image  plane  produces 
an  image,  Cimage  identical  to  £.  (that  is,  the  angle  and  intercept  remain  9  and  yo 
respectively  -  see  Figure  6).  A  typical  output  from  a  CCD  (charge  coupled  device) 
video  camera  (the  only  type  being  considered  here)  is  an  analog  signal  constructed 
from  a  number  of  intensity  readings  from  discrete  pixels  in  a  rectangular  grid.  For 
this  analysis,  we  assume  that  the  image  has  been  thresholded  and  transformed  into 
a  binary  array  of  square  elements  with  length 

For  geometric  reasons,  when  9  €  [f  >  one  pixel  per  row  is  illuminated  and 
when  9  e  [-J.f],  only  one  pixel  for  every  column  is  illuminated.  We  define  an 
illuminated  pixel  as  the  one  in  a  particular  row  (or  column)  whose  border  circum¬ 
scribes  the  longest  segment  of  Cimage-  Although  this  definition  loses  some  positional 
information  from  the  array,  it  simplifies  the  subsequent  analysis.  Later  in  this  sec¬ 
tion  corrections  from  grey  level  information  are  suggested  which  preserve  more  of 
the  available  positional  information. 

The  sensor  data  used  for  curve  fitting  is  a  list  of  pixel  center  coordinates  for 
the  illuminated  elements  (fi,y,)  i  =  l,...N.  A  “best  fit”  orientation  and  intercept 
(9  and  ifo  respectively)  may  be  found  in  the  least  squares  sense  by  minimizing  the 
sum  of  the  squares  of  the  distances  from  the  best  fit  line  to  data  points  (x,,  ff,)  with 


Figure  7:  Probabilistic  location  of  points  on  a  line.  a.  Intersection  of  a  column  of 
pixels  with  a  line  in  the  range  ^  €  {— |,  |).  b.  Probability  density  function  for  the 
location  of  the  y  coordinate  of  Cimag*  measured  from  the  center  of  the  illuminated 
pixel. 


distance  measured  one  of  three  ways: 

A:  Minimize  distances  parallel  to  the  y  axis. 
B:  Minimize  distances  parallel  to  the  x  axis. 


C:  Minimize  distances  perpendicular  to  the  line. 

Each  of  these  gives  a  slightly  different  result  for  B  and  jJd  and  the  best  choice  is  not 
immediately  apparent. 

Selection  of  the  most  appropriate  metric  to  minimize  depends  upon  the  gross 
orientation  of  £.  As  a  result  of  our  definition  of  an  illuminated  pixel,  (one  pixel 
for  every  y  coordinate  is  illuminated  when  B  G  and  one  pixel  for  every  x 

coordinate  is  illuminated  when  B  G  [— |,  |]),  we  can  choose  an  x  coordinate  in  the 
first  case  which  is  deterministic  (£,-  will  always  correspond  to  a  point  on  the  line 
within  that  pixel  -  see  Figure  7a),  and  a  y  coordinate  in  the  second  case  which 
is  deterministic.  With  these  choices  of  coordinates,  we  should  minimize  errors  in 
probabilistic  coordinates  y,-  in  the  first  case  (metric  A)  and  x,-  in  the  second  case 
(metric  B). 

In  the  subsequent  amalysis  B  is  assumed  to  lie  in  the  interval  (— j,  |].  The  least 

A 

squares  estimates  for  orientation  B  and  intercept  are  found  by  minimizing  errors 
parallel  to  the  y  axis  [1,4] 


B  =  arctan 


y.Xf  - 


ih  = 


*»•  N 


(19) 


N 


(20) 


wh«re  (x{,y{)  are  the  coordinates  of  points  lying  on  If  i^tUi)  were  known 

precisely,  the  least  squares  estimates  would  be  identical  to  the  actual  line  parame¬ 
ters;  however,  in  the  interval  $  6  {— f ,  f  ],  only  precise  x,-  values  are  available. 

We  are  considering  the  case  where  z,  is  deterministic  and  pi  is  probabilistic  and 
we  desire  a  metric  for  the  confidence  of  the  estimated  line  parameters  0  and 
calculated  from  the  measurements  (f^,  Pi).  We  shaJl  derive  the  variance  of  0  and 
(<ri  and  respectively)  as  a  function  of  all  (f,,  s^)’s  and  the  variance  of  p,  (o^.)- 
Each  random  variable  p,-  may  be  written  as 


Ifc  =  y.  +  SVi 


(21) 


where  is  deterministic  and  £pj  is  a  random  variable  with  zero  mean.  At  first  it 
might  appear  that  the  ^p^’s  are  highly  correlated  since  they  all  lie  on  the  same  line; 
however,  some  authors  [22,27]  have  suggested  that  independence  between  £p,’s  is 
a  good  assumption  for  certain  cases.  In  the  case  of  the  straight  line,  the  degree 
of  correlation  depends  upon  the  orientation  of  the  line.  At  some  orientations,  the 
values  of  £p,-  may  change  in  an  unrelated  fashion  (slightly  correlated)  while  at  other 
orientations,  the  values  of  ^p{  may  exhibit  a  periodic  pattern  (highly  correlated). 
We  make  the  initial  assumption  that  the  fp,’s  are  independent  and  a  supposition 
that  this  might  not  be  valid  for  lines  at  certain  orientations. 

If  the  maximum  absolute  value  of  the  jp,’s  are  small,  (19)  and  (20)  can  be  lin¬ 
earized  about  the  points  (£.,  jft)  by  taking  the  Taylor  series  expansion,  and  keeping 
the  first  two  terms 


*  ^/d0\ 

w  -h  2  (  —  )  6yi 

(22) 

+  hi 

(23) 

where  (£:),  . 

V«Vi ;  {f,  ,#;) 


is  the  partial  derivative  with  respect  to  p,  evaluated  at  (Z,,p,)  and 


0  =  arctan 


Eili  tfjXj  - 

z,’ 


(24) 


-tangE.^1^. 
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For  all  Syi  independent  of  one  another  [10] 

.S-v  * 


(25) 


2  , 
i=i 


(26) 


Substituting  (19)  and  (20)  into  (26)  and  (27)  respectively  then  making  the  assump¬ 
tion  that  all  Sf/i  have  the  same  variance  er^,  and  setting  x,-  to  we  obtain 


"  I  E." .  ’  - 


S 


where 


Eili  tiiXi  - 


pAT  *■•)* 

1^=1 »»  ff 

Equations  (28)  and  (29)  give  the  variances  of  the  fitted  line  parameters  as  a  function 
of  the  data  points  (z,-,  |7,)  and  the  variance  of  the  location  of  the  y  coordinate  of 
the  line  in  any  column  containing  an  illuminated  pixel  (similar  results  are  obtained 
in  (41). 

If  columns  ^  have  an  illuminated  pixel  for  all  t  =  1, . . .  iV  then  the  expressions 
for  the  variance  of  9  and  ^  become 

"I  ^  (1  +  m»)»  N’  -N 


"ft" 


12z* 

jV»- 


^em  ^  J 


where  Zem  =  *  is  the  calculated  center  of  the  line  segment  in  the  z  direction. 

These  results  describe  the  accuracy  with  which  a  line  may  be  found  from  a  least 
squares  technique  as  a  function  of  the  approximate  slope  (m),  the  orthographic 
projection  of  the  length  of  the  line  (in  pixels)  onto  the  z  axis  (AT),  the  center  of  the 
line  in  the  z  direction  (zem)i  and  the  variance  of  the  location  of  the  y  coordinate 
for  any  one  column  of  the  image  (a*),  al  does  not  depend  on  the  position  of  the 
line  segment  in  the  pixel  grid,  while  depends  on  the  distance  of  the  center  of 
the  line  segment  from  the  y  axis.  For  a  line  (zero  width),  the  probability  density 
function  for  mutually  independent  ^y,-  is  a  uniform  distribution  one  pixel  in  width 
with  a  (see  Figure  7b).  The  standard  deviations  ay  and  are 

plotted  in  Figure  8  as  a  function  of  the  number  of  points  N  for  a  line  positioned 
near  9  =  0  and  with  Xem  = 


Figure  8:  Standard  deviations  for  orientation  and  y  intercept  estimates  as  a  function 
of  the  number  of  illuminated  pixels. 

The  above  analysis  is  valid  for  a  grey  scale  images  of  finite  width  lines  as  well 
as  binary  images  of  thin  lines.  If  is  permitted  to  take  on  subpixel  values,  and 
grey  scale  levels  are  used  in  an  intensity  weighted  calculation  for  the  center  of  the 
line,  y.  becomes  a  better  estimate  of  the  actual  value  y<  (aj  would  be  smaller  -  see 
(17)).  The  reduction  in  a*  is  a  function  of  the  width  of  the  line  and  the  resolution 
(in  intensity  measurements)  of  the  camera. 

In  support  of  the  theoretical  analysis,  computer  simulations  and  experimental 
tests  were  performed.  In  both  studies,  sample  statistics  from  30  trials  were  calcu¬ 
lated  and  used  as  a  basis  for  comparisons.  Figure  9  shows  the  results  of  the  study 
with  statistics  generated  in  each  of  eighteen  equally  spaced  orientations  from  zero 
to  forty-five  degrees.  The  sample  standard  deviations  for  each  set  of  thirty  trials 
(each  trial  is  at  a  different  random  orientation  within  a  2.5  degree  range)  with  a 
line  128  pixels  long  are  plotted  for  each  interval  along  with  the  theoretical  result, 
(30).  A  plot  of  the  accuracy  of  fitted  parameters  of  lines  from  45  to  90  degrees 
from  metric  B  would  appear  as  the  mirror-image  of  Figure  9  about  the  45  degree 
position.  Singularities  in  A0  occur  at  slopes  near  0,  j,  j,  |,  and  1  (shown  in 
figure).  In  these  areas,  measurements  of  the  location  and  orientation  of  Cimage 
not  as  accurate  because  there  is  more  space  for  C  to  translate  and  rotate  before 
pixels  are  caused  to  change  state,  (see  Figure  10).  The  theoretical  result  does 
not  predict  these  singularities  because  the  mutual  independence  assumption  is  not 
valid  there.  The  simulation  and  experimental  sample  standard  deviations  at  all 
orientations  are,  however,  theoretically  bounded  by  an  analysis  assuming  perfect 
correlation  [18|. 


f)'fcorefital  Si.  Dov.  (Perlectly  Coirelaled) 
-  i  hooretical  3f.  Dov.  (Uncorielated) 
o  Sinulaa'on  Sample  St.  Dev. 

K  £?<iici  imeiual  Sample  SI.  Dev. 


OrionUttion  (Dagf<c:es) 

l  east  Squares  Fitted  Orientation  Error 


Figure  9:  Accuracy  of  the  least  squares  fitting  routine  for  discretized  lines  as  a 
function  of  their  orientation.  Theoretical,  computer  simulation  and  experimental 
results  are  shown. 
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Figure  10:  Range  of  motions  of  lines  before  change  in  state  of  sensor  occurs. 


5  Using  Redundant  S«ised  Information 

Techniques  for  combining  similar  information  from  a  number  of  sources  or  mul¬ 
tiple  measurements  from  the  same  source  may  be  used  to  improve  the  accuracy  of 
a  single  measurement  [6,7,11,33].  The  equivalent  of  four  or  more  line  segments  may 
be  generated  on  a  polyhedral  feature  either  by  using  two  or  more  images  with  the 
part  or  light  plane  displaced  slightly,  or  by  using  multiple  light  planes  and  a  single 
image.  In  addition  to  allowing  a  more  accurate  calculation  of  the  location  of  a  part 
feature,  a  system  of  multiple  light  planes  has  the  advantage  of  having  illumination 
sources  which  are  not  in  a  single  plane;  thus,  it  may  be  possible  to  generate  longer 
line  segments  or  have  line  segments  in  more  desirable  locations  with  respect  to  the 
part  feature.  In  fact,  some  features  may  be  localized  with  a  number  of  light  planes 
but  not  with  a  single  light  plane. 

5.1  Orientation  Estimation  for  Polyhedral  Features  Without  Measure¬ 
ment  Covariance  Information 

This  technique  does  not  take  the  quality  of  different  measurements  (covariance) 
into  account.  A  similar  problem  is  addressed  by  Faugeras  and  Hubert  [12,13,14] 
who  find  the  best  fit  orientation  of  an  object  from  a  number  of  sensed  normal 
vectors.  We  were  not  able  to  use  their  technique  since  our  ir  easurement  data  is 
in  the  form  of  line  segments  on  the  fac^  of  the  feature.  With  each  additional  line 
segment,  there  is  an  additional  equation  associated  with  (10)  and  we  may  minimize 
the  sum  of  the  squares  of  the  dot  products 

min  ^(/<  •  qn,q*)*.  (32) 

i 

A  value  for  q  which  minimizes  this  quartic  expression  may  be  found  using  a  nu¬ 
merical  technique. 

5.2  Optimal  Linear  Estimation  of  Orientation  for  a  Polyhedral  Feature 

In  this  technique,  the  covariance  of  the  measurements  are  taken  into  account 
and  a  linear  estimator  is  used  to  estimate  the  state  (position)  of  the  object. 

5.2.1  Representation  of  Rotations 

For  the  subsequent  analysis  we  use  an  Euler  angle  description  of  rotations  be¬ 
cause  the  number  of  variables  needed  to  describe  a  rotation  is  equal  to  the  number 
of  degrees  of  freedom.  The  Euler  angle  convention  is  shown  in  Figure  11.  A  coor- 


Figure  11:  Euler  angles  used  in  the  least  squares  estimate  for  the  orientation  of  a 
polyhedral  feature. 

dinate  frame  located  at  Euler  angles 


is  first  rotated  about  the  base  system  z  axis  by  angle  9  then  about  its  own  x  axis 
by  angle  then  about  its  own  z  axis  by  angle 

5.3.2  Obtaining  an  Optimal  Linear  Estimate  from  Image  Line  Slopes 

We  assume  that  orientation  estimates  expressed  in  Euler  angle  notation, 
are  available.  In  general,  these  may  be  obtuned  from  three  appropriately  chosen 
line  segments. 

CLi  —  ^,s)t  ♦  =  1, . . . ,  A/  (33) 

where  /<  is  a  vector  function,  5i,2,and5«^  are  the  slopes  of  the  three  line  seg¬ 
ments  from  the  sensor,  and  Af  is  the  number  of  different  estimates.  We  wish  to 
combine  these  estimates  to  arrive  at  the  most  accurate  estimate  possible,  a,  given 
the  expected  errors  of  the  measurements. 

The  functions  /,■  are  functions  of  the  calibration  variables  and  part  model  pa¬ 
rameters  as  well  as  the  slopes  of  the  line  segments.  For  simplicity,  only  the  errors 
in  slopes  are  considered  in  the  following  Kalman  filter  design.  If  the  magnitude  of 
the  unmodeled  uncertainties  are  large  enough,  imprecise  results  may  be  generated. 

In  general  the  functions  /,-  in  (33)  are  non-linear  and  we  cannot  directly  apply 
linear  optimal  estimation  theory  [16].  We  can,  however,  assume  that  errors  in 


slope  measurements,  SSij,  are  small  and  linearize  (33).  Taking  the  Taylor  series 
expanmn  and  keeping  terms  of  first  order  or  less 


Si  fa 


^^y^SSii 


(34) 


where  I  takes  on  the  appropriate  values  for  the  lines  used  in  the  estimates  and  6  Sij 
is  a  randmn  variable  which  is  the  difference  between  the  actual  slope  for  line  /  and 
the  measured  slope.  Neglecting  higher  order  terms,  (34)  in  matrix  form  is 

Si  =  d^  +  Ks,iSSi  (35) 

where 

and  SSi  =  ^  SSij 

I  SS/,s  . 

The  partial  derivatives  may  be  calculated  directly  if  an  analytical  expression  is 
available  for  /,-.  If  /,-  involves  a  numerical  technique,  the  partial  derivatives  may  be 
approximated  by 

.  .  /»(5t,l, . . .  ,  (5ij  +€),...,  5i,s) 

--  'f--  ■'  '  ■  "  T-  -  r.  ..  .  r... 

dSij 

where  e  is  a  small  number  whose  addition  to  Sij  will  produce  a  result  in  the  nu¬ 
merical  calculation  significantly  larger  than  the  numerical  error. 

Rearranging  terms,  (35)  becomes 


a/7  a/7  a/7 
a5..i’as..,’as..5 


KjJa.  =  Ks,^a.  -  «5,,  1  =  1,...,M. 


•-Ijs 


(37) 


All  M  equations  in  (37)  may  be  combined  into  a  single  equation  by  adding  the  rows 
of  each  vector  term  corresponding  to  the  same  line  segment 


zs  =  HsS-hvs 


(38) 


where 

=  H5  =  5:Ki;, 

t  i 

Vs  =  S'Si. 
i 

The  sums  are  taken  over  identical  lines  only  so  that  (38)  has  the  same  number  of 
rows  as  independent  line  slopes  in  the  image. 

Equation  (38)  is  in  the  linear  optimal  estimation  equation  form  [16];  thus,  the 
maximum  likelihood  estimate  for  the  Euler  angles  is 


(39) 


where  C5  is  the  covariance  matrix  of  the  noise  vector  05.  This  matrix  is  diagonal 
with  elements  equal  to  the  variances  of  the  slopes  of  the  respective  lines.  These 
variances  are  calculated  from  (30). 

The  minimum  variance  Bayesian  estimate  of  the  Euler  angles  given  an  a  priori 
estimate  a—  and  initial  covariance  matrix  P~  is  [16] 

a  =  a-([p-|-‘  +  Hf Ci»(xs  -  HgcT)  (40) 

which  is  a  discrete  Kalman  filter.  This  recursive  estimator  may  be  applied  to 
position  information  from  fimdamentally  different  sources  so  long  as  a  measure  of 
the  expected  accuracy  (covariance  matrix)  of  the  information  is  available. 

6  Results  of  Prototype  Sensor  Accuracy  Tests 

A  prototype  light-stripe-part-position  sensor  mounted  in  a  test  bed  arrangement 
is  shown  in  Figure  12.  The  sensor  consists  of  an  MOS-type  camera  with  320  x  244 
pixels  and  a  laser-cylindrical  lens  light  source.  The  test  bed  has  three  translational 
stages  and  three  rotational  stages. 

The  measurement  accuracy  of  the  test-bed-mounted  sensor  was  studied  by  mea¬ 
suring  the  comer  feature  of  a  cube,  Figure  13.  Comer  location  measurements  were 
compared  to  the  settings  on  the  six  stages  of  the  test  apparatus.  These  measure¬ 
ments  were  relative  to  an  arbitrary  reference  location.  It  was  not  necessary  to 
obtain  absolute  measurements  since  relative  accuracy  is  the  specification  required 
to  determine  sensor  performance  in  an  assembly  environment  where  all  measure¬ 
ments  are  relative  to  the  robot  frame. 

The  relative  accuracy  in  locating  the  comer  of  the  cube  using  information  from 
two  light  planes  was  also  studied.  The  maximum  likelihood  technique  developed  in 
Section  5  was  used  to  obtain  estimates  of  the  orientation  of  the  cube. 

6.1  Test  Procedure 

The  test  procedure  for  determining  the  accuracy  of  the  sensor  entailed  locating 
the  cube  at  a  home  position  then  moving  one  of  the  six  positioning  stages  (call  this 
the  test  direction)  a  small  amount.  An  image  of  the  cube  was  recorded  and  the 
same  stage  was  moved  the  same  amount  once  again.  This  was  repeated  a  number 
of  times.  The  positions  of  the  cube  corner  from  a  series  of  images  were  compared 
to  the  apparatus  positions  and  sample  statistics  were  calculated  from  differences 
between  stage  motion  and  sensed  comer  position.  The  test  procedure  was  repeated 
for  all  six  stages.  The  rotational  degrees  of  freedom  in  sts^e  motion,  Zrot,  ^rot,  and 
i{>rot  are  defined  in  Figtire  13.  Measurements  were  made  in  the  9,  and  V'  degrees 
of  freedom,  which  are  the  Euler  angles  of  the  cube.  Figure  11  shows  the  convention 


12:  Test  bed  used  for  sensor  accuracy  studies 


Figure  13:  Test  cube  mounted  on  rotational  stages. 


Table  2;  Results  of  the  part  position  sensor  accuracy  tests.  Each  number  is  the 
sample  standard  deviation  from  21  images.  Mean  sample  standard  deviations  are: 
translation  -  .002  inches  (.055  mm),  rotation  -  .095  degrees  (.0015  radians). 

used.  Since  the  cube  was  fairly  accurately  aligned  to  the  axes  of  the  positioner,  4>rot 
corresponds  closely  to  (f>  and  i^rot  corresponds  closely  to  0. 

The  range  of  motions  for  the  tests  were  0.5  inches  (12.7  mm)  in  each  translational 
degree  of  freedom  and  8.5  degrees  (.15  radians)  for  the  0roe  rotation  and  0rot  rotation 
and  20  degrees  (.35  radians)  for  the  rotation.  In  general,  measurements  near 
the  center  of  the  ranges  of  motion  were  more  accurate  than  those  near  the  ends. 
This  was  because  line  segments  on  some  cube  faces  become  smaller  as  the  cube  was 
moved  from  a  central  position  under  the  laser  stripe. 

6.2  Single  Light  Plane  Test  Results 

Table  2  shows  the  results  of  the  tests  for  motions  of  each  of  the  six  degrees  of 
freedom.  The  blank  spaces  in  the  table  correspond  to  degrees  of  freedom  which 
were  non-stationary  or  were  not  linearly  related  to  the  test  direction  motion.  The 
mean  of  the  sample  standzu-d  deviations  are  .002  inches  (.05  mm)  in  translation  and 
0.095  degrees  (0.0015  radians)  in  rotation.  The  translational  error  of  .002  inches 
(.05  mm)  is  equivalent  to  about  ^  of  a  camera  pixel.  Assuming  the  distribution  of 
errors  are  Gaussian,  98.8  percent  of  the  measurements  (5  standard  deviations)  will 
be  within  an  accuracy  of  .010  inches  (.25  mm)  and  .5  degrees  (.007  radians).  Each 
measurement  of  the  position  of  the  cube  took  about  2.5  seconds.  Much  of  this  time 
was  during  an  image  convolution  step  used  to  more  accurately  locate  the  centers 
of  the  line  segments  [17|. 

The  repeatability  of  the  sensor  was  extremely  high  (about  .0001  inches  or  .0025 
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Figure  16:  Maximum  likelihood  estimates  of  rotation  angles  of  test  cube  from  pairs 
of  angles.  Plots  are  for  Euler  angles  ^  and  if. 


some  angles  seems  to  be  an  imprecise  estimate.  This  u  most  likely  due  to  the  cou¬ 
pling  between  the  Euler  angles  and  the  unmodeled  calibration  imcertmnties.  The 
maximum  likelihood  estimates  from  pairs  of  angles  are  shown  in  Figure  16.  Here 
the  maximum  likelihood  estimate  is  between  the  two  independent  laser  estimates. 
The  estimate  nears  one  of  the  single  laser  estimates  when  that  one  is  more  reliable 
(longer  line  segments)  than  the  other.  The  estimate  always  uses  the  combination  of 
the  measurements  which  reduces  the  overall  error.  Since  calibration  errors  were  not 
taken  into  account,  the  estimate  weigl»  the  two  calculations  equally  for  equal  length 
line  segments.  Because  the  maximiun  likelihood  estimate  performs  a  weighted  av¬ 
erage  of  the  two  estimates,  and  the  two  estimates  have  different  means,  the  sample 
standard  deviation  of  the  manmum  likelihood  estimate  is  not  always  less  than  the 
smaller  sample  standard  deviation  of  the  single  laser  estimates;  although,  the  tests 
showed  that  it  was  usually  very  close  to  the  lower  standard  deviation  of  the  two. 

7  Summary  and  Conclusions 

A  light  stripe  based  vision  sensor  for  locating  polyhedral  features  of  parts  from  a 
single  video  frame  has  been  developed.  The  system  locates  features  to  an  accuracy 


of  .002  inches  (.05  nun  or  about  ^  of  a  camera  pixel)  in  translation  and  about  0.1 
degrees  (0.0015  radians)  in  rotation.  The  measurements  take  about  2.5  seccmds. 
This  may  be  speeded  up  to  about  1  second  if  a  different  hardware  image  convolver 
is  used. 

The  calculation  for  locating  general  polyhedra  requires  a  numerical  computa¬ 
tion.  If  the  three  planes  of  the  feature  are  orthogonal,  a  closed  form  solution  for 
the  feature  location  may  be  used.  The  accuracy  of  the  feature  locating  technique 
depends  on  the  lengths  of  the  line  segments  generated  from  the  intersection  of  the 
light  plane  and  the  part’s  planar  surfaces.  The  accuracy  of  fitting  lines  to  data  from 
quantized  images  was  found  as  a  function  of  the  length,  orientation,  and  position 
of  the  segments. 

Two  techniques  for  using  redundant  data  from  multiple  light  planes  were  dis¬ 
cussed.  One  technique  may  be  used  to  combine  measurement  data  from  sources 
in  addition  to  the  light  stripe  sensor.  The  covariance  of  a  measurement  from  the 
light  stripe  system  is  a  fimction  of  the  variance  in  locating  the  line  segments  in  the 
image;  thus,  the  optimal  linear  estimate  weighs  those  estimates  with  the  longest 
length  segments  more  than  those  with  shorter  length  segments. 

The  sensing  system  is  useful  for  a  number  of  industrial  tasks  and  was  demon¬ 
strated  to  be  practical  for  detecting  misalignments  between  parts  prior  to  their 
assembly  by  a  manipulator. 

Non-polyhedral  features  generate  non-linear  curves  of  intersection  and  a  method 
of  determining  the  covariance  matrix  for  the  parameters  of  more  general  curves  is 
required  if  they  are  to  be  incorporated  into  the  optimal  estimation  technique  pre¬ 
sented  in  Section  5.  More  work  is  required  in  sensing  non-polyhedral  features 
and  analyzing  non-linear  curves  in  light  stripe  images.  Bolles  and  Fischler  [9]  and 
Lozano-P4rez,  Grimson,  and  White  [25]  report  difiSculty  in  fitting  conic  sections 
to  light  stripe  data.  Ellipse  fitting,  from  intersection  curves  of  cylindrical  features, 
gives  significantly  less  accurate  results  than  the  line  fitting  techniques  for  polyhe¬ 
dral  features.  We  are  investigating  multiple  light  plane  intersections  of  cylindrical 
features  which  produce  additional  constraints  which  could  be  used  in  the  fitting 
procedure  to  give  better  results  (see  [25]  for  a  discussion  of  multiple  light  plane 
techniques  for  fitting  cylinders). 
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